Thanks for the review. We see the memorization floor flag and understand the concern.

A few questions to make sure we comply correctly:

1. What TTT configuration is considered legal? Is it strictly 1 epoch (single-pass, score-then-train per chunk)? Or is there a specific epoch/adaptation limit?

2. Is the concern about the number of epochs, or about scoring below a BPB floor? If we ran 1 epoch and happened to score below 0.95, would that also be flagged?

3. Is the merged PR [record bpb=1.195] sliding window + LoRA TTT #77 pattern the gold standard? Single pass, score chunk, train on it, next chunk, reset between documents?

We're happy to resubmit with single-epoch backward-looking TTT to stay within whatever the organizers consider legal. Our architecture + quantization alone puts us at ~1.18 BPB pre-TTT, and we believe even single-pass TTT will put us below the current SOTA.

We want to compete on the merits, not on a gray area.

Thanks for the requests for clarification! I think the problem with this submission is around line 950 in the TTT scheme: the code evals a doc, then trains on it for multiple epochs, and the final loss that the model reports is this loss-post-doc-training, not the initial eval loss before you adapted the weights. I believe this means this scheme trains on the eval tokens, and is therefore invalid.

1. I can't speak to all possible implementations of TTT, but I definitely treat multi-epoch training with a lot more suspicion than single-epoch, plainly due to the much higher risk of unintentional eval information leakage.
2. I can't see what review you're replying to for some reason, but my concerns are specifically with the code in train_gpt.py, not with a specific loss value or abstractly, the number of epochs.
3. Yeah, it's certainly a valid way of doing it, so I would go for that implementation first and then try to improve if you're not SOTA.

Closing for now, but feel free to reopen once you have fixed these, if the result is still SOTA (specifically, if it beats the just-merged SOTA, PR #549, or whatever future SOTA supersedes it by the time you have a new submission ready).

You're right — the multi-epoch approach trains on eval tokens across epochs. By the final epoch (the one whose scores we report), the LoRA has already been trained on every token for N-1 complete passes. That is training on eval data.

Would a single-epoch TTT (score-then-train, each token scored exactly once before any training on it) be considered valid? In single-pass, the LoRA adapts to the document's distribution but never scores tokens it has already trained on.

If single-epoch is legal, we'd like to resubmit with ttt_epochs=1. If all TTT is ruled out, we'll submit our non-TTT baseline.